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IN THE CLAIMS 
Please amend the claims as follows: 

Claim 1 (Original): A virtual viewpoint image generation method comprising: 
a step of obtaining plural images of an object taken by plural cameras; 
a step of determining a virtual viewpoint that is a position from which the object is 
viewed; and 

a step of generating a virtual viewpoint image that is an image of the object viewed 
from the virtual viewpoint based on the obtained images of the object, 
the step of generating the virtual viewpoint image comprising: 

a step 1 of setting projection planes having a multi-layered structure; 
a step 2 of obtaining each corresponding point, on the images of the object, 
corresponding to a projection point of a projection plane; 

a step 3 of determining color information or brightness information of each 
projection point based on color information or brightness information of corresponding 
points; 

a step of calculating, for each of the projection points overlapping when 
viewed from a reference viewpoint in a space, a degree of probability that the object exists at 
a distance corresponding to a position of the projection point based on a degree of correlation 
of the corresponding points or neighborhoods of the corresponding points; 

a step 5 of performing mixing processing on color information or brightness 
information of reference points overlapping when viewed from the virtual viewpoint 
according to the degree of probability of existence of the object so as to determine color 
information or brightness information of each pixel of the virtual viewpoint image; and 

a step 6 of repeating steps from the step 1 to the steps 5 for every point 
corresponding to pixels of the virtual viewpoint image. 
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Claim 2 (Original): The virtual viewpoint image generation method as claimed in 
claim 1, the step 3 comprising: 

mixing the color information or the brightness information of the corresponding 
points or selecting the color information or the brightness information of one corresponding 
point from the color information or the brightness information of the corresponding points. 

Claim 3 (Original): The virtual viewpoint image generation method as claimed in 
claim 1 or 2, the step 4 or the step 5 comprising: 

a step of setting, in each reference point on the projection plane, transparency having 
plural gradations from transparent to opaque by converting the degree of probability that the 
object exists; and 

the step 5 comprising: 

performing the mixing processing according to the transparency instead of the degree 
of probability that the object exists. 

Claim 4 (Original): The virtual viewpoint image generation method as claimed in 
claim 3, the mixing processing of the step 5 comprising: 

processing projection points successively from a projection point far from the virtual 
viewpoint to a projection point near the virtual viewpoint, 

wherein color information or brightness information obtained by the mixing 
processing up to a projection point is obtained by calculating interior division between color 
information or brightness information at the projection point and color information or 
brightness information obtained by the mixing processing up to a previous projection point in 
a ratio according to the transparency. 
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Claim 5 (Currently Amended): The virtual viewpoint image generation method as 
claimed in claim 1 any on e of claims 1 4 , wherein[[:]] 

projection planes specific to each camera taking each image of the object are set in the 

step 1 ; 

the color information or the brightness information of the step 3 are determined only 
by using color information or brightness information of the corresponding points of the 
images of the object taken by the plural cameras; 

the degree of probability that the object exists in step 4 is calculated using, as the 
reference viewpoint, a viewpoint of the camera specific to the projection plane to which the 
projection point belongs; and 

correction is performed based on position relationship between the virtual viewpoint 
and each reference viewpoint in the mixing processing of the color information or the 
brightness information. 

Claim 6 (Currently Amended): A virtual viewpoint image generation apparatus,, 
comprising: 

[[an]] object image obtaining means for obtaining plural images of an object taken by 
plural cameras; 

[[a]] virtual viewpoint determination means for determining a virtual viewpoint that is 
a position from which the object is viewed; and 

[[an]] image generation means for generating a virtual viewpoint image that is an 
image of the object viewed from the virtual viewpoint based on the obtained images of the 
object, 

the image generation means comprising: 
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[[a]] projection plane determination means for determining projection planes 
having a multi-layered structure; 

[[a]] reference viewpoint determination means for determining a position of 
the reference viewpoint; 

[[a]] texture array keeping means for keeping an array of texture images to be 
mapped to the projection planes; 

[[a]] corresponding point matching processing means for associating parts, in 
the images of the object, on which the same region of the object appears with each other; 

[[a]] color information determination means for determining color 
information or brightness information in the array of the texture images by performing 
mixing processing on the images of the object; 

[[an]] existence probability information determination means for calculating a 
degree of probability that the object exists at a distance corresponding to a position of the 
projection point in the array of the texture images based on the processing result of the 
corresponding point matching processing means; 

[[a]] rendering means for rendering the projection planes viewed from the 
virtual viewpoint based on the color information or the brightness information determined by 
the color information determination means and the existence probability determined by the 
existence probability determination means. 

Claim 7 (Currently Amended): The virtual viewpoint image generation apparatus as 
claimed in claim 6, the existence probability information determination means comprising: 

[[a]] means for setting, in each reference point on the projection plane, transparency 
having plural gradations from transparent to opaque by converting the degree of possibility 
that the object exists; 



7 



Application No. 10/561,344 

Reply to Office Action of July 23, 2008 

wherein the rendering means performs rendering using the transparency instead of the 

degree of possibility that the object exists. 

Claim 8 (Currently Amended): The virtual viewpoint image generation apparatus as 
claimed in claim 7, the rendering means comprising: 

[[a]] means for processing projection points successively from a projection point far 
from the virtual viewpoint to a projection point near the virtual viewpoint, 

wherein color information or brightness information obtained by the mixing 
processing up to a projection point is obtained by calculating interior division between color 
information or brightness information at the projection point and color information or 
brightness information obtained by the mixing processing up to a previous projection point in 
a ratio according to the transparency. 

Claim 9 (Currently Amended): The virtual viewpoint image generation apparatus as 
claimed in claim 6 any on e of claims 6 8 , wherein[[:]] 

the projection plane determination means determines projection planes specific to 
each camera taking each image of the object; 

the color information determination means determines the color information or the 
brightness information only by using color information or brightness information of the 
corresponding points of the images of the object taken by the plural cameras; 

the existence probability information determination means calculates the degree of 
probability that the object exists by using, as the reference viewpoint, a viewpoint of the 
camera specific to the projection plane to which the projection point belongs; and 

the rendering means includes a means for performing correction based on position 
relationship between the virtual viewpoint and each reference viewpoint. 
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Claim 10 (Currently Amended): A virtual viewpoint imag e g e n e ration program 
causing a comput e r to perform: A computer readable storage medium encoded with a 
computer readable virtual viewpoint image generation program configured to cause an 
information processing apparatus to execute a method, the method comprising: 

a step of obtaining plural images of an object taken by plural cameras; 

a step of determining a virtual viewpoint that is a position from which the object is 
viewed; and 

a step of generating a virtual viewpoint image that is an image of the object viewed 
from the virtual viewpoint based on the obtained images of the object, 
the step of generating the virtual viewpoint image comprising: 

a step 1 of setting projection planes having a multi-layered structure; 
a step 2 of obtaining each corresponding point, on the images of the object, 
corresponding to a projection point of a projection plane; 

a step 3 of determining color information or brightness information of each 
projection point based on color information or brightness information of corresponding 
points; 

a step 4 of calculating, for each of the projection points overlapping when 
viewed from a reference viewpoint in a space, a degree of probability that the object exists at 
a distance corresponding to a position of the projection point based on a degree of correlation 
of the corresponding points or neighborhoods of the corresponding points; 

a step 5 of performing mixing processing on color information or brightness 
information of reference points overlapping when viewed from the virtual viewpoint 
according to the degree of probability of existence of the object so as to determine color 
information or brightness information of each pixel of the virtual viewpoint image; and 
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a step 6 of repeating steps from the step 1 to the steps 5 for every point 
corresponding to pixels of the virtual viewpoint image. 

Claim 1 1 (Canceled). 

Claim 12 (Currently Amended): An image generation method^ comprising: 
a step of obtaining images of an object taken from different viewpoints; 
a step of obtaining a three-dimensional shape of the object based on the images; and 
a step of generating an image of the object viewable vi e w e d from a viewpoint of an 
observer based on the obtained three-dimensional shape of the object, 

the step of obtaining the three-dimensional shape of the object comprising: 

a step of setting projection planes having a multi-layered structure in a virtual 
three-dimensional space; 

a step of determining a reference viewpoint for obtaining the three- 
dimensional shape of the object; 

a step of determining color information or brightness information of projection 
points that are points on the projection planes based on color information or brightness 
information of corresponding points, on the obtained images, corresponding to the projection 
points; 

a step of calculating correlation degrees among corresponding points 
corresponding to the projection points; 

a step of calculating, for each of the projection points overlapping when 
viewed from the reference viewpoint, an existence probability that is a probability that the 
object exists at the projection point based on the correlation degree of each projection point; 
the step of calculating the correlation degree comprising: 
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a step of preparing plural groups of camera sets each being a group of some 
viewpoints selected from the plural viewpoints; and 

a step of obtaining the correlation degree from the corresponding points on 
images included in each camera set, 

the step of determining the existence probability comprising: 

a step of calculating the existence probability based on the correlation degree 
of each projection point obtained for each camera set; and 

a step of determining the existence probability of each projection point by 
performing integrating processing for the existence probability determined for each camera 
set. 

Claim 13 (Currently Amended): The image display method as claimed in claim 12, 
the step of calculating the existence probability based on the correlation degree of each 
projection point obtained for each camera set comprising: 

a step of calculating an evaluation reference value based on the correlation degree 
d e gr eee of each projection point calculated for each camera set; 

a step of calculating a distribution function of the existence probability by performing 
statistical processing on the evaluation reference value of each projection point calculated for 
each camera set; and 

a step of determining the existence probability of each projection point based on the 
distribution function of the existence probability. 

Claim 14 (Currently Amended): The image generation method as claimed in claim 12 
or 1 3, the step of generating the image of the object viewable vi e w e d from the viewpoint ef 
the obs e rv e r comprising: 
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mixing color information or brightness information of the projection points 

overlapping when viewable vi e w e d from the viewpoint of th e obs e rv e r in a ratio of the 

existence probability to determine color information or brightness information of each point 

on the image to be generated so as to generate a two-dimensional image. 

Claim 15 (Currently Amended): The image generation method as claimed in claim 12 
or 13, the step of generating the image of the object viewable vi e w e d from the viewpoint ef 
th e obs e rv e r comprising: 

a step of setting plural image generation planes at positions having different depths 
when viewable vi e w e d from the viewpoint of th e obs e rv e r ; and 

a step of converting the color information or brightness information and the existence 
probability of each projection point into color information or brightness information and a 
brightness distribution coefficient on each image generation plane based on position 
relationship among projection points overlapping when viewable vi e w e d from the viewpoint 
of th e obs e rver and points on each image generation plane. 

Claim 16 (Currently Amended): An image generation^ comprising: 

[[an]] object image obtaining means for obtaining images of an object taken from 
different viewpoints; 

[[a]] three-dimensional shape obtaining means for obtaining a three-dimensional 
shape of the object based on the images; and 

[[an]] object image generation means for generating an image of the object viewable 
view e d from a viewpoint of an obs e rv e r based on the obtained three-dimensional shape of the 
object, 

the three-dimensional shape obtaining means comprising: 
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[[a]] means for setting projection planes having a multi-layered structure in a virtual 
three-dimensional space; 

[[a]] means for determining a reference viewpoint for obtaining the three- 
dimensional shape of the object; 

[[a]] means for determining color information or brightness information of projection 
points that are points on the projection planes based on color information or brightness 
information of corresponding points, on the obtained images, corresponding to the projection 
points; 

[[a]] means for calculating correlation degrees among corresponding points 
corresponding to the projection points; 

[[a]] means for determining, for each of the projection points overlapping when 
viewed from the reference viewpoint, an existence probability that is a probability that the 
object exists at the projection point based on the correlation degree of each projection point; 

the means for calculating the correlation degree comprising: 

[[a]] means for preparing plural groups of camera sets each being a group of some 
viewpoints selected from the plural viewpoints; and 

[[a]] means for obtaining the correlation degree from the corresponding points on 
images included in each camera set, 

the means for determining the existence probability comprising: 

[[a]] means for calculating the existence probability based on the correlation degree 
of each projection point obtained for each camera set; and 

[[a]] means for determining the existence probability of each projection point by 
performing integrating processing for the existence probability determined for each camera 
set. 
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Claim 17 (Currently Amended): The image generation apparatus as claimed in claim 
16, the means for calculating the existence probability based on the correlation degree of each 
projection point obtained for each camera set comprising: 

[[a]] means for calculating an evaluation reference value based on the correlation 
value of each projection point calculated for each camera set; 

[[a]] means for calculating a distribution function of the existence probability by 
performing statistical processing on the evaluation reference value of each projection point 
calculated for each camera set; and 

[[a]] means for determining the existence probability of each projection point based 
on the distribution function of the existence probability. 

Claim 18 (Currently Amended): The image generation apparatus as claimed in claim 
16 or 1 7, wherein the means for generating the image of the object viewable vi e w e d from the 
viewpoint of th e obs e rv e r is a means for: 

mixing color information or brightness information of the projection points 
overlapping when viewable vi e w e d from the viewpoint of th e obs e rv e r in a ratio of the 
existence probability to determine color information or brightness information of each point 
on the image to be generated so as to determine a two-dimensional image. 

Claim 19 (Currently Amended): The image generation apparatus as claimed in claim 
16 or 1 7, the means for generating the image of the object viewable view e d from the 
viewpoint of the observ e r comprising: 

[[a]] means for setting plural image generation planes at positions having different 
depths when viewable vi e w e d from the viewpoint of th e obs e rver ; and 
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[[a]] means for converting the color information or brightness information and the 

existence probability of each projection point into color information or brightness 

information and a brightness distribution coefficient on each image generation plane based on 

position relationship among projection points overlapping viewable vi e w e d from the 

viewpoint of the observer and points on the image generation planes. 

Claim 20 (Currently Amended): An imag e g e n e ration program causing a comput e r to 
p e rform: A computer readable storage medium encoded with a computer readable image 
generation program configured to cause an information processing apparatus to execute a 
method, the method comprising: 

a step of obtaining images of an object taken from different viewpoints; 

a step of obtaining a three-dimensional shape of the object based on the images; and 

a step of generating an image of the object viewable vi e w e d from a viewpoint e£an 
ob se rv e r based on the obtained three-dimensional shape of the object, 

the step of obtaining the three-dimensional shape of the object comprising: 

a step of setting projection planes having a multi-layered structure in a virtual 
three-dimensional space; 

a step of determining a reference viewpoint for obtaining the three- 
dimensional shape of the object; 

a step of determining color information or brightness information of projection 
points that are points on the projection planes based on color information or brightness 
information of corresponding points, on the obtained images, corresponding to the projection 
points; 

a step of calculating correlation degrees among corresponding points 
corresponding to the projection points; 
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a step of calculating, for each of the projection points overlapping when 
viewed from the reference viewpoint, an existence probability that is a probability that the 
object exists at the projection point based on the correlation degree of each projection point; 
the step of calculating the correlation degree comprising: 

a step of preparing plural groups of camera sets each being a group of some 
viewpoints selected from the plural viewpoints; and 

a step of obtaining the correlation degree from the corresponding points on 
images included in each camera set, 

the step of determining the existence probability comprising: 

a step of calculating the existence probability based on the correlation degree 
of each projection point obtained for each camera set; and 

a step of determining the existence probability of each projection point by 
performing integrating processing for the existence probability determined for each camera 
set. 

Claim 21 (Canceled). 

Claim 22 (Original): An image generation method comprising: 

a step of obtaining images of an object taken by changing focusing distance; 

a step of obtaining a virtual viewpoint that is a viewpoint from which the object 

appearing in the images is viewed; 

a step of obtaining a three-dimensional shape of the object based on the images; and 
a step of generating an image of the object viewed from the virtual viewpoint based 

on the obtained three-dimensional shape of the object, 

the step of obtaining the three-dimensional shape of the object comprising: 
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a step of setting projection planes having a multi-layered structure in a virtual 
three-dimensional space; 

a step of determining a reference viewpoint for obtaining the three- 
dimensional shape of the object; 

a step of determining color information or brightness information of projection 
points, that are points on the projection planes, based on color information or brightness 
information of corresponding points, on the obtained images, corresponding to the projection 
points; 

a step of determining focusing degrees of the projection points based on 
focusing degrees of the corresponding points corresponding to the projection points; 

a step of calculating, for each of the projection points overlapping when 
viewed from the reference viewpoint, an existence probability that is a probability that the 
object exists at a distance corresponding to a position of the projection point based on the 
focusing degree of each projection point; 

the step of generating the image of the object viewed from the virtual viewpoint 
comprising: 

mixing color information or brightness information of the projection points 
overlapping when viewed from the virtual viewpoint in a ratio corresponding to the existence 
probability to determine color information or brightness information of each point of the 
image to be generated. 

Claim 23 (Original): The image generation method as claimed in claim 22, the step of 
obtaining the three-dimensional shape of the object or the step of generating the image of the 
object viewed from the virtual viewpoint comprising: 
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a step of setting transparency having plural gradations from transparent to opaque on 
each projection point based on the existence probability of the projection points overlapping 
when viewed from the reference viewpoint or the virtual viewpoint; 

the step of generating the image of the object viewed from the virtual viewpoint 
comprising: 

mixing the color information or the brightness information of the projection 
points overlapping when viewed from the virtual viewpoint in a ratio according to the 
transparency set based on the existence probability so as to determine the color information 
or brightness information of each point of the image to be generated. 

Claim 24 (Original): The image generation method as claimed in claim 23, 
the step of generating the image of the object viewed from the virtual viewpoint 
comprising: 

mixing the color information or the brightness information for projection points 
successively from a projection point far from the virtual viewpoint to a projection point near 
the virtual viewpoint, 

wherein color information or brightness information obtained by the mixing 
processing up to a projection point is obtained by calculating interior division between color 
information or brightness information at the projection point and color information or 
brightness information obtained by the mixing processing up to a previous projection point in 
a ratio according to the transparency. 

Claim 25 (Currently Amended): An image generation apparatus^ comprising: 
[[an]] object image obtaining means for obtaining images of an object taken by 
changing focusing distance; 
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[[a]] virtual viewpoint setting means for setting a virtual viewpoint that is a viewpoint 
from which the object appearing in the images is viewed; 

[[a]] three-dimensional shape obtaining means for obtaining a three-dimensional 
shape of the object based on the images; and 

[[a]] rendering means for generating an image of the object viewed from the virtual 
viewpoint based on the obtained three-dimensional shape of the object, 

the three-dimensional shape obtaining means comprising: 

[[a]] means for setting projection planes having a multi-layered structure in a 
virtual three-dimensional space; 

[[a]] means for determining a reference viewpoint for obtaining the three- 
dimensional shape of the object; 

[[a]] means for determining color information or brightness information of 
projection points, that are points on the projection planes, based on color information or 
brightness information of corresponding points, on the obtained images, corresponding to the 
projection points; 

[[a]] means for determining focusing degrees of the projection points based 
on focusing degrees of the corresponding points corresponding to the projection points; 

[[a]] means for determining, for each of the projection points overlapping 
when viewed from the reference viewpoint, an existence probability that is a probability that 
the object exists at a distance corresponding to a position of the projection point based on the 
focusing degree of each projection point; 
the rendering means comprising: 

[[a]] means for mixing color information or brightness information of the 
projection points overlapping when viewed from the virtual viewpoint in a ratio 
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corresponding to the existence probability to determine color information or brightness 
information of each point of the image to be generated. 

Claim 26 (Currently Amended): The image generation apparatus as claimed in claim 
25, the three-dimensional shape obtaining means or the rendering means comprising: 

[[a]] means for setting transparency having plural gradations from transparent to 
opaque on each projection point based on the existence probability of the projection points 
overlapping when viewed from the reference viewpoint or the virtual viewpoint; and 

the rendering means comprising: 

[[a]] means for mixing the color information or the brightness information of the 
projection points overlapping when viewed from the virtual viewpoint in a ratio according to 
the transparency set based on the existence probability so as to determine the color 
information or brightness information of each point of the image to be generated. 

Claim 27 (Original): The image generation apparatus as claimed in claim 26, 
the rendering means comprising: 

a means for mixing the color information or the brightness information for projection 
points successively from a projection point far from the virtual viewpoint to a projection 
point near the virtual viewpoint, 

wherein color information or brightness information obtained by the mixing 
processing up to a projection point is obtained by calculating interior division between color 
information or brightness information at the projection point and color information or 
brightness information obtained by the mixing processing up to a previous projection point in 
a ratio according to the transparency. 
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Claim 28 (Currently Amended): An image generation program for causing a 
comput e r to perform:: A computer readable storage medium encoded with a computer 
readable image generation program configured to cause an information processing apparatus 
to execute a method, the method comprising: 

a step of obtaining images of an object taken by changing focusing lengths; 

a step of obtaining a virtual viewpoint that is a viewpoint from which the object 
appearing in the images is viewed; 

a step of obtaining a three-dimensional shape of the object based on the images; and 

a step of generating an image of the object viewed from the virtual viewpoint based 
on the obtained three-dimensional shape of the object, 

the step of obtaining the three-dimensional shape of the object comprising: 

a step of setting projection planes having a multi-layered structure in a virtual 
three-dimensional space; 

a step of determining a reference viewpoint for obtaining the three- 
dimensional shape of the object; 

a step of determining color information or brightness information of projection 
points, that are points on the projection planes, based on color information or brightness 
information of corresponding points, on the obtained images, corresponding to the projection 
points; 

a step of determining focusing degrees of the projection points based on 
focusing degrees of the corresponding points corresponding to the projection points; 

a step of calculating, for each of the projection points overlapping when 
viewed from the reference viewpoint, an existence probability that is a probability that the 
object exists at a distance corresponding to a position of the projection point based on the 
focusing degree of each projection point; 
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the step of generating the image of the object viewed from the virtual viewpoint 
comprising: 

mixing color information or brightness information of the projection points 
overlapping when viewed from the virtual viewpoint in a ratio corresponding to the existence 
probability to determine color information or brightness information of each point of the 
image to be generated. 

Claim 29 (Canceled). 

Claim 30 (Currently Amended): An image generation methods comprising: 
a step of obtaining images of an object taken under different conditions; 
a step of obtaining a three-dimensional shape of the object based on the images; and 
a step of generating an image of the object viewable vi e w e d from a viewpoint of an 
obs e rv e r based on the obtained three-dimensional shape of the object, 

the step of obtaining the three-dimensional shape of the object comprising: 

a step of setting projection planes having a multi-layered structure in a virtual 
three-dimensional space; 

a step of determining a reference viewpoint for obtaining the three- 
dimensional shape of the object; 

a step of determining color information or brightness information of projection 
points, that are points on the projection planes, based on color information or brightness 
information of corresponding points, on the obtained images, corresponding to the projection 
points; 
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a step of determining, for each of the projection points overlapping when 
viewed from the reference viewpoint, an existence probability that is a probability that a 
surface of the object exists at the projection point; 

the step of determining the existence probability comprising: 

a step of calculating an evaluation reference value of each projection point 
from image information of the corresponding points; 

a step of performing statistical processing on the evaluation reference value of 
each projection point; and 

a step of calculating the existence probability of each projection point based 
on the evaluation reference value on which the statistical processing has been performed. 



Claim 31 (Original): The image generation method as claimed in claim 30, the step of 
obtaining the plural images comprising: 

obtaining images by taking the object from different viewpoints; 
the step of determining the existence probability comprising: 

a step of obtaining correlation degrees among corresponding points 
corresponding to the projection points; 

a step of calculating the evaluation reference value based on the correlation 
degree of each projection point; 

a step of performing the statistical processing on the evaluation reference 

value; and 

a step of calculating the existence probability of each projection point based 
on the evaluation reference value on which the statistical processing has been performed. 
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Claim 32 (Original): The image generation method as claimed in claim 30, the step of 
obtaining the plural images comprising: 

obtaining the images of the object taken from a viewpoint by changing focusing 
distance; 

the step of determining the existence probability comprising: 

a step of calculating focusing degrees of the projection points from focusing 

degrees of the corresponding points corresponding to the projection points; 

a step of calculating the evaluation reference value based on the focusing 

degree of each projection point; 

a step of performing the statistical processing on the evaluation reference 

value; and 

a step of calculating the existence probability of each projection point based 
on the evaluation reference value on which the statistical processing has been performed. 

Claim 33 (Original): The image generation method as claimed in claim 30, the step of 
obtaining the plural images comprising: 

obtaining images of the object taken from plural viewpoints, and images of the object 
taken from equal to one or more viewpoints among the plural viewpoints by changing 
focusing distance; 

the step of determining the existence probability comprising: 

a step of obtaining correlation degrees between the projection points and 
corresponding points on the plural images of different viewpoints; 

a step of obtaining a first evaluation reference value based on the correlation 
degrees of each projection point; and 
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a step of performing the statistical processing on the first evaluation reference 

value; 

a step of calculating focusing degrees of the projection points based on 
focusing degrees of the corresponding points on the images of different focusing distances; 

a step of calculating a second evaluation reference value based on the focusing 
degrees of each projection point; and 

a step of calculating the existence probability of each projection point based 
on the first evaluation reference value and the second evaluation reference value on which the 
statistical processing has been performed. 

Claim 34 (Currently Amended): The image generation method as claimed in any one 
of claims 30-33, the step of generating the image of the object viewable vi e w e d from the 
viewpoint of th e observ e r comprising: 

mixing color information or brightness information of the projection points 
overlapping when viewable view e d from the viewpoint of th e obs e rv e r in a ratio of the 
existence probability to determine color information or brightness information of each point 
on the image to be generated so as to determine a two-dimensional image. 

Claim 35 (Currently Amended): The image generation method as claimed in any one 
of claims 30-33, the step of generating the image of the object viewable vi e w e d from the 
viewpoint of th e obs e rv e r comprising: 

a step of setting plural image generation planes at positions having different depths 
when viewable vi e w e d from the viewpoint of th e obs e rv e r ; and 

a step of converting the color information or brightness information and the existence 
probability of each projection point into color information or brightness information and a 
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brightness distribution coefficient on each image generation plane based on position 
relationship among projection points overlapping viewable vi e w e d from the viewpoint of th e 
obs e rv e r and points on the image generation planes. 

Claim 36 (Currently Amended): An image generation apparatus,, comprising: 
an object image obtaining means for obtaining images of an object taken under 
different conditions; 

[[an]] object shape obtaining means for obtaining a three-dimensional shape of the 
object based on the images; and 

[[an]] object image generation means for generating an image of the object viewable 
v i e w e d fr om a viewpoint of an obs e rv e r based on the obtained three-dimensional shape of the 
object, 

the object shape obtaining means comprising: 

[[a]] means for setting projection planes having a multi-layered structure in a 
virtual three-dimensional space; 

[[a]] means for determining a reference viewpoint for obtaining the three- 
dimensional shape of the object; 

[[a]] means for determining color information or brightness information of 
projection points, that are points on the projection planes, based on color information or 
brightness information of corresponding points, on the obtained images, corresponding to the 
projection points; 

[[a]] means for determining, for each of the projection points overlapping 
when viewed from the reference viewpoint, an existence probability that is a probability that 
a surface of the object exists at the projection point; 

the means for determining the existence probability comprising: 
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[[a]] means for calculating an evaluation reference value of each projection 
point from image information of the corresponding points; 

[[a]] means for performing statistical processing on the evaluation reference 
value of each projection point; and 

[[a]] means for calculating the existence probability of each projection point 
based on the evaluation reference value on which the statistical processing has been 
performed. 

Claim 37 (Currently Amended): The image generation apparatus as claimed in claim 
36, wherein the object image obtaining means obtains images of the object taken from 
different viewpoints; 

the means for determining the existence probability comprising: 

[[a]] means for obtaining correlation degrees among corresponding points 
corresponding to the projection points; 

[[a]] means for calculating the evaluation reference value based on the correlation 
degree of each projection point; 

[[a]] means for performing the statistical processing on the evaluation reference value; 

and 

[[a]] means for calculating the existence probability of each projection point based on 
the evaluation reference value on which the statistical processing has been performed. 

Claim 38 (Currently Amended): The image generation apparatus as claimed in claim 
36, wherein the object image obtaining means obtains the images of the object taken from a 
viewpoint by changing focusing distance; 

the means for determining the existence probability comprising: 
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[[a]] means for calculating focusing degrees of the projection points from focusing 
degrees of the corresponding points corresponding to the projection points; 

[[a]] means for calculating the evaluation reference value based on the focusing 
degree of each projection point; 

[[a]] means for performing the statistical processing on the evaluation reference 
value; and 

[[a]] means for calculating the existence probability of each projection point based on 
the evaluation reference value on which the statistical processing has been performed. 

Claim 39 (Currently Amended): The image generation apparatus as claimed in claim 
36, wherein the object image obtaining means obtains images of the object taken from plural 
viewpoints, and images of the object taken from equal to one or more viewpoints among the 
plural viewpoints by changing focusing distance; 

the means for determining the existence probability comprising: 

[[a]] means for obtaining correlation degrees between the projection points and 
corresponding points on the plural images having different projection points; 

[[a]] means for obtaining a first evaluation reference value based on the correlation 
degrees of each projection point; and 

[[a]] means for performing the statistical processing on the first evaluation reference 

value; 

[[a]] means for calculating focusing degrees of the projection points based on 
focusing degrees of the corresponding points on the images of different focusing distances 
taken from a viewpoint; 

[[a]] means for calculating a second evaluation reference value based on the focusing 
degrees of each projection point; and 
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[[a]] means for calculating the existence probability of each projection point based on 
the first evaluation reference value and the second evaluation reference value on which the 
statistical processing has been performed. 

Claim 40 (Currently Amended): The image generation apparatus as claimed in any 
one of claims 36-39, the means for generating the image of the object viewable vi e w e d from 
the viewpoint of the obs e rver comprising: 

[[a]] means for mixing color information or brightness information of the projection 
points overlapping when viewable vi e w e d from the viewpoint of the observ e r in a ratio of the 
existence probability to determine color information or brightness information of each point 
on the image to be generated so as to determine a two-dimensional image. 

Claim 41 (Currently Amended): The image generation apparatus as claimed in any 
one of claims 36-39, the means for generating the image of the object viewable vi e w e d from 
the viewpoint of the obs e rv e r comprising: 

[[a]] means for setting plural image generation planes at positions having different 
depths when viewable viewed from the viewpoint of the obs e rv e r ; and 

[[a]] eans for converting the color information or brightness information and the 
existence probability of each projection point into color information or brightness 
information and a brightness distribution coefficient on each image generation plane based on 
position relationship among projection points overlapping viewable vi e w e d from the 
viewpoint of the observ e r and points on the image generation planes. 

Claim 42 (Currently Amended): An image g e n e ration program causing a comput e r to 
p e rform: A computer readable storage medium encoded with a computer readable image 



29 



Application No. 10/561,344 

Reply to Office Action of July 23, 2008 

generation program configured to cause an information processing apparatus to execute a 
method, the method comprising: 

a step of obtaining images of an object taken under different conditions; 

a step of obtaining a three-dimensional shape of the object based on the images; and 

a step of generating an image of the object viewable vi e w e d from a viewpoint of an 
observ e r based on the obtained three-dimensional shape of the object, 

the step of obtaining the three-dimensional shape of the object comprising: 

a step of setting projection planes having a multi-layered structure in a virtual 
three-dimensional space; 

a step of determining a reference viewpoint for obtaining the three- 
dimensional shape of the object; 

a step of determining color information or brightness information of projection 
points, that are points on the projection planes, based on color information or brightness 
information of corresponding points, on the obtained images, corresponding to the projection 
points; 

a step of determining, for each of the projection points overlapping when 
viewed from the reference viewpoint, an existence probability that is a probability that a 
surface of the object exists at the projection point; 

the step of determining the existence probability comprising: 

a step of calculating an evaluation reference value of each projection point 
from image information of the corresponding points; 

a step of performing statistical processing on the evaluation reference value of 
each projection point; and 

a step of calculating the existence probability of each projection point based 
on the evaluation reference value on which the statistical processing has been performed. 
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Claim 43 (Canceled). 

Claim 44 (Currently Amended): A three-dimensional image display method^ 
comprising: 

a step of obtaining images of an object taken under different conditions; 

a step of obtaining a three-dimensional shape of the object based on the images; 

a step of setting a viewpoint position such that from which an obs e rv e r obs e rv e s 
plural image display planes existing at different depth positions are viewable wh e n vi e wed 
from th e observer ; 

a step of generating two-dimensional images to be displayed on each image display 
plane based on the obtained three-dimensional shape of the object; and 

a step of presenting a three-dimensional image of the object by displaying the 
generated two-dimensional images on each display plane, 

the step of obtaining the three-dimensional shape of the object comprising: 

a step of setting projection planes having a multi-layered structure in a virtual 
three-dimensional space; 

a step of determining a reference viewpoint for obtaining the three- 
dimensional shape of the object; 

a step of determining color information or brightness information of projection 
points, that are points on the projection planes, based on color information or brightness 
information of corresponding points, on the obtained images, corresponding to the projection 
points; 
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a step of determining, for each of the projection points overlapping when 
viewed from the reference viewpoint, an existence probability that is a probability that a 
surface of the object exists at the projection point; 

the step of generating the two-dimensional images comprising: 

converting the color information or the brightness information and the 
existence probability of the projection point into color information or brightness information 
and existence probability of each display point that is a point on the image display plane 
corresponding to the projection plane on which the projection point exists so as to generate 
the two dimensional images; 

the step of presenting the three-dimensional image of the object comprising: 

displaying the color information or brightness information on each display 
point with brightness according to the existence probability. 

Claim 45 (Original): The three-dimensional image display method as claimed in 
claim 44, the step of obtaining the plural images comprising: 

obtaining images of the object taken from different viewpoints, 
the step of determining the existence probability comprising: 

a step of calculating correlation degrees among corresponding points 
corresponding to the projection points; 

a step of determining, for projection points overlapping when viewed from the 
reference viewpoint, the existence probability of each projection point based on the 
correlation degree of each projection point. 

Claim 46 (Original): The three-dimensional image display method as claimed in 
claim 44, the step of obtaining the plural images comprising: 
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obtaining images of the object taken from a viewpoint by changing focusing distance, 

the step of determining the existence probability comprising: 

a step of calculating focusing degrees of the projection points from focusing 

degrees of corresponding points corresponding to the projection points; 

a step of determining, for projection points overlapping when viewed from the 

reference viewpoint, the existence probability of each projection point based on the focusing 

degree of each projection point. 

Claim 47 (Original): The three-dimensional image display method as claimed in 
claim 44, the step of obtaining the plural images comprising: 

obtaining images of the object taken from plural viewpoints, and images of the object 
taken from equal to one or more viewpoints among the plural viewpoints by changing 
focusing distance; 

the step of determining the existence probability comprising: 

a step of obtaining correlation degrees between the projection points and 
corresponding points on the plural images of different viewpoints; 

a step of calculating focusing degrees of the projection points based on 
focusing degrees of the corresponding points on the images, having different focusing 
distance, of each viewpoint; and 

a step of determining, for projection points overlapping when viewed from the 
reference viewpoints, the existence probability of each projection point based on the 
correlation degree and the focusing degree of each projection point. 

Claim 48 (Currently Amended): A three-dimensional image display apparatus 
comprising: 
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an object image obtaining means for obtaining images of an object taken under 
different conditions; 

[[a]] three-dimensional shape obtaining means for obtaining a three-dimensional 
shape of the object based on the images; 

an observer vi e wpoint s e tting means for setting a viewpoint position such that from 
which an observ e r obs e rv e s plural image display planes existing at different depth positions 
are viewable when view e d from the observ e r ; 

[[a]] two-dimensional image generation means for generating two-dimensional 
images to be displayed on each image display plane based on the obtained three-dimensional 
shape of the object; and 

wherein the three-dimensional image display apparatus presents a three-dimensional 
image of the object by displaying the generated two-dimensional images on each display 
plane, 

the three-dimensional shape obtaining means comprising: 

[[a]] means for setting projection planes having a multi-layered structure in a 
virtual three-dimensional space; 

[[a]] means for determining a reference viewpoint for obtaining the three- 
dimensional shape of the object; 

[[a]] means for determining color information or brightness information of 
projection points, that are points on the projection planes, based on color information or 
brightness information of corresponding points, on the obtained images, corresponding to the 
projection points; 

[[a]] means for determining, for each of the projection points overlapping 
when viewed from the reference viewpoint, an existence probability that is a probability that 
a surface of the object exists at the projection point; 
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the two-dimensional image generation means comprising: 

[[a]] means for converting the color information or the brightness information 
and the existence probability of the projection point into color information or brightness 
information and existence probability of each display point that is a point on the image 
display plane corresponding to the projection plane on which the projection point exists to 
generate the two dimensional images; 

wherein the three-dimensional image display apparatus displays the color 
information or brightness information on each display point with brightness according to the 
existence probability. 

Claim 49 (Currently Amended): The three-dimensional image display apparatus as 
claimed in claim 48, wherein the object image obtaining means is a means for obtaining 
images of the object taken from different viewpoints, 

the means for determining the existence probability comprising: 

[[a]] means for calculating correlation degrees among corresponding points 
corresponding to the projection points; 

[[a]] means for determining, for projection points overlapping when viewed from the 
reference viewpoint, the existence probability of each projection point based on the 
correlation degree of each projection point. 

Claim 50 (Currently Amended): The three-dimensional image display apparatus as 
claimed in claim 48, wherein the object image obtaining means is a means for obtaining 
images of the object taken from a viewpoint by changing focusing distance, 

the means for determining the existence probability comprising: 
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[[a]] means for calculating focusing degrees of the projection points from focusing 
degrees of corresponding points corresponding to the projection points; and 

[[a]] means for determining, for projection points overlapping when viewed from the 
reference viewpoint, the existence probability of each projection point based on the focusing 
degree of each projection point. 

Claim 51 (Currently Amended): The three-dimensional image display apparatus as 
claimed in claim 48, wherein the object image obtaining means is a means for obtaining 
images of the object taken from plural viewpoints, and images of the object taken from equal 
to one or more viewpoints among the plural viewpoints by changing focusing distance; 

the means for determining the existence probability comprising: 

[[a]] means for obtaining correlation degrees between the projection points and 
corresponding points on the plural images of different viewpoints; 

[[a]] means for calculating focusing degrees of the projection points based on 
focusing degrees of the corresponding points on the images, having different focusing length, 
of each viewpoint; and 

[[a]] means for determining, for projection points overlapping when viewed from the 
reference viewpoints, the existence probability of each projection point based on the 
correlation degree and the focusing degree of each projection point. 

Claim 52 (Currently Amended): A thr e e dim e nsional imag e display program cau s ing 
a comput e r to p e rform: A computer readable storage medium encoded with a computer 
readable three-dimensional image display program configured to cause an information 
processing apparatus to execute a method, the method comprising: 

a step of obtaining images of an object taken under different conditions; 
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a step of obtaining a three-dimensional shape of the object based on the images; 

a step of setting a viewpoint position such that from which an observer observes 
plural image display planes existing at different depth positions are viewable when vi e w e d 
from th e obs e rv e r ; 

a step of generating two-dimensional images to be displayed on each image display 
plane based on the obtained three-dimensional shape of the object; and 

a step of presenting a three-dimensional image of the object by displaying the 
generated two-dimensional images on each display plane, 

the step of obtaining the three-dimensional shape of the object comprising: 

a step of setting projection planes having a multi-layered structure in a virtual 
three-dimensional space; 

a step of determining a reference viewpoint for obtaining the three- 
dimensional shape of the object; 

a step of determining color information or brightness information of projection 
points, that are points on the projection planes, based on color information or brightness 
information of corresponding points, on the obtained images, corresponding to the projection 
points; 

a step of determining, for each of the projection points overlapping when 
viewed from the reference viewpoint, an existence probability that is a probability that a 
surface of the object exists at the projection point; 

the step of generating the two-dimensional images comprising: 

a step of converting the color information or the brightness information and 
the existence probability of the projection point into color information or brightness 
information and existence probability of each display point that is a point on the image 
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display plane corresponding to the projection plane on which the projection point exists to 
generate the two dimensional images; 

the step of presenting the three-dimensional image of the object comprising: 

a step of displaying the color information or brightness information on each 
display point with brightness according to the existence probability. 

Claim 53 (Canceled). 
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